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ABSTRACT 



| The use of behaviorally anchored rating scales (BARS) 



as the basis of an assessment system that was designed ^toimprove 
academic department chairpersons in a college of arts and sciences is 
described. Twenty-eight:\ faculty members, two from each department, 
were asked to identify evaluative dimensions for assessing / 
chairperson performance and to provide critical/behavioral incidents 
that would demonstrate poor, adequate, and good performance on each 
of the 11 dimensions. Aftet review by a panel, 236 incidents were 
identified and were rewritten into an expectations format. Forty-two 
faculty were asked to make two judgments for the list of items. The 
first judgment required categorization of the 245 items into /ll 
performance dimensions. The second judgment required placement of 
each item on a seven-point scale based on the level of performance 
indicated. Seven scales $hat were generated were distributed to all 
faculty of 14 academic departments to, generate primary information 
for a chairperson performance assessment. System application and 
examination of behavj orally anchored rating scale results, interviews 
with faculty and, key administrative staff, and self-reports of 
chairpersons will be components of the chairperson performance 
assessment. Perspectives oh administrative performance evaluation and 
features of BARS are also considered. It is suggested that the 
generation of BARS itself helps to clarify goals, and that its 
participative and collaborative Characteristics help to ensure that 
values of the collegial body are being considered. A bibliography is 
appended. (SH) ' 
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BASING PERFORMANCE ASSESSMENT ON 
BEHAVIORALLY ANCHORED RATING. SCALES 
IN COLLEGIATE ORGANIZATIONS 

Introduction 

The assessment of managerial performance in colleges and universities 
has not enjoyed wi3e success and little research has been reported regarding , 
assessment of performance of academic administrators, in particulars Assess- 
ment as- tised in this paper includes activities carried out to enable the 
academic manager to improve his or her performance, as well as activities 
undertake* to determine quality of performance from the judgmental perspective 
of a senior manager or administrator. . \ 

The purpose of this paper is to describe how an organization has used\ 
iJehaviorally anchored rating ^scales (bArS) as the basis of an assessment system 
intended to provide information for self-improvement for academic department 
chaxrpersons in a college of arts and sciences. 

Perspective 

Cohen and Majch (1974) refer to the college or university organization 
as an organized anarchy which exhibits the following general properties: 
(1) problematic goals 1 (ones that are vague or in dispute) ; (2) unclear tech- 
nology (its own processes are not well understood) ; and, (3) fluid participation, 
where organization participants are free to vary in the amount of effort and 

time they devote to the organization, over time. 

\ 

Within such a settifiij the assessment of performance is difficult because 
most models of assessment and evaluation are premised on theories of manage- 
ment and administration whick assume the presence. of well-defined goals as 
well as substantial participant\involvement in the activities of the organization 



(Cohen and March, p. 4). ' The methods of assessment used in the present study 
take special cognizance of the characteristics of the college organization 
with attempts to validate performance by way of multiple methods of assess- 

»*»'., i • 

ment. 

Using multiple met-hods to assess performance has been labeled triangulation 
(Green and Stone, 1977) . Triangulation allows one to appraise the same variables 
from several aspects which may permit confirmation, substantiation, and 
verification of observations. Thus, multiple methods may compensate for / 
inadequacies or weaknesses of a single method of assessment. In ^the present 
investigation the administrators of a college of arts and sciences used the 
methods of structured interviews with faculty, interviews with key administrative 
staff, self-reports and self-ratings\>f performance by the subjects and, 
finally, the empirically-derived, behaviorally anchored rating scales for use 
with all college of arts and sciences faculty. A thorough assessment effort 
— ^ldTTs-a-pdsItxv^aspect-, lend 'credence to the overall act of assessment 
in the eyes of the subjects, academic department chairpersons. 

The academic department chairperson occupies a pivotal position in the 

t 

development and implementation of academic programs because he/she interacts 
^with many institutional offices and members on a day-to-day basis regarding 
• the"*del^very of educational services and because the proper and smooth operation 
of the academic program is dependent upon the performance of the department 
head or chairperson'. The role and functions of chairpersons is examined 
extensively in the literature of higher education (Brown, 1977) . 

Evaluation of the administrative performance of the chairperson is largely 
a matter of .concern internal to the university. Evaluation of chairpersons 
will most likely result in the opportunity for improvement in performance 



through assessments oi strengths and needs, and through an awareness of 
perceptions of persons wit^h whom the chairperson .works. 

Systematic evaluatior^ of administrative performance in universities has 
not enjoyed wide success. 1 Farmer (1979) states' that evaluation of adminis- 
trative functions is a highly politicized process that contains little 
objectivity. Dressel (197G) warns tfiat evaluation of administrative functions 
is most difficult in higher! education organizations because few people agree 
on what criteria define success in administration. Booth (1978, p. 80' reports 
that many ca£e studi<?s^ have shown the capacity of chairpersons to make improve- 
ments in' the operation of academic departments and he goes on to say that 
wore systematic attention t6 Ithe evaluation of chairpersons would probably 
produce a goad many administrative imjDjov^inents, 



The role and functions 
fully examined in a monograph 



the academic department chairperson were care- 
by Waltzer (1975) . Through detailed interviews 

or present and former chairpersons and a large sample of academic and service 

'I 

administrators, the stqdy sought to present a practical look at the expectations 
and realities of the job of Jhe academic department chairperson as it is, 
currently, Waltzer found thai the job carries' little formal authority, and 
that the authority that is posited in the job derives from what university 
and divisional administrators and department faculty allow in particular 
circumstances. Further, the chairpersons reported that increased bureaucrati- 
zation and the prevailing styles of governance by councils, committees, and 
the like, spread authority to many other places and diminishes the authority 
available to the chairpersons. 

A kind of condition or status accrues to the chairperson in which he or 
she has the responsibility for making the contradictory; elements of effective, 



efficient management and maintenance of coilegial community function as a vital 

» 

enterprise. Waltzer clarifies this condition by pointing out that the chair- 
persons must: (1) manage administrative directives and coilegial decision 
making; (2) retain the friendship and respect of their colleagues while 
implementing policies that directly affect faculty; and, (3) accept responsibility 
for all departmental affairs but be one among equals in their departments 
(Waltzer, p. ] 4) . This condition supports the need for performance assessment 
whiSh is highly objective and free from political influences. t % % 

A question arises as to the value and importance of the chair position. 

- L 

In some colleges and universities the chairperson role is seen as ohe that is 
reluctantly held and/or one that may be rotated among senior members of a 
department. In many colleges and universities departmental responsibilities 
are shared in a highly coilegial environment where the role and influence of 
the chair is minimal. In other organizations the chairperson is a powerful 
member of the organization with much potential-i-trfiuence-.- 

It seems reasonable to assume that in the 1980 , s and beyond, chairpersons 
are likely to be expected^by colleagues (including superiors) to be skilled 
in .managerial functions, and, chairpersons are more likely to be evaluated 
according to managerial performance criteria than bn'the basis of purely 
academic performance criteria. 

Millet (1978) regards the chairperson as a program planner and program 
manager. In the role* of program planner the chairperson is concerned with 

providing department leadership in addressing such matters as student numbers, 

* if 

student, quality, student advising, student* performance and accomplishment of 
degree requirements. In the manager rolo the chairperson is working on tasks 
having to do with faculty personnel actions (recruitment, promotion, terure, 



separation), faculty budget actions, (compensation, supplies, equipment, travel, 
departmental support) , support personnel actions (recruitment, work assignment, 
.etcO# faculty facilities (offices, classrooms, laboratories), and work sched- 
uling (Millet, p. 53)* 

With the variety ofTunctions, activities, and actors' interacting with the 
role- of chairperson, the need for multiple methods of assessment to verify 
performance is made prominent. It was generally anticipated that the kinds of 
task activities listed above would emerge from the assessment methods as the 
major domains or dimensions of performance. These activities certainly are not 
the exclusive listing of all such activities. Other tasks and functions such 
as representation of the department to external publics and one's personal 
professional performance as a scholar and teacher may be regarded as vital 
functions to be evaluated. 

It appears that careful definition and a high degree of objectivity are 
characteristics which need to be part of the evaluation of academic depart- 
ment "chairpersons. In a recent publication, Nordvall (1979, p. 14) points 
out that rating scales of performance related to characteristics such as those 
represented generally by leadership, interpersonal relationships, basic under- 
lying traits, and commitment to institution seem to enjoy wide use. As typically 
developed and implemented (arid som^ imes borrowed from other institutions) 
the, rating scale can be a device fraught with problems. Many- evaluation 
rating forms tend to be disorganized and ambiguous. Evaluation rating^scales 
may contain global behavior measures and vague traij descriptions. Rating 

scales are available for the evaluation of chairpersons, but according to 

x /' 

Hodgkinson (1978, p. 110) , "they are mostly opinion Surveys and do not represent 
"behavioral consensus on what excellent\performance means. 11 
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Methods/Techniques 

In order to overcpme the shortcomings of rating scales and in order to 
✓ « 
establish a firm undergirding for thw development and implementation of structured 

interviews and self-report guides, the methodoligy of the behaviorally anchored 

rating scale was chosen for use. . 

Based upon the work of Smith and Kendall (1963), and Harari and Zedeck 

(1973), Blood (1974), (1979) has demonstrated how a performance appraisal , . 

technique, behaviorally anchored sating scales (BARS) , positively responds to 

many of the problems of evaluation identified above. ^The features of BARS 

are as follows: 

1. A population of would-be raters are ^as^^* individually , to 

- £ \ 

write descriptions of behavioral episodes ^/hat define scale 
points. Hence, performance is defined in terms of observable 
behaviors by members of the population who later will do the 



rating, Borman and Dunnette (1975) address the point that 
the methodology has good potential for overcoming or reducing 
many of the errors often encountered in job performance, rating 
systems- The involvement of superiors, subordinates, and/or 
job incumbents in all phases of the development of job behavior 
observation scales should enhance and facilitate the choosing 

of job dimensions and behavior examples that-are readilv ^ 

i 

understood and accepted by the persons asked to make the j 
performance ratings. They indicate that by collecting critical* 
incidents about job performance and then using them to define 
dimensions and to anchor different levels of performance on 
each. dimension, the method should also help to decrease the 
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semantic ambiguities that tend to , be prevalent ii%„many 
performance rating systems. .Because levels of performance, 
are better defined one should anticipate decreased error 
attributed to leniency; because performance dimensions are 
better specified decreased halo effects should obtain; 
because raters are likely to be more attentive £o the rating 
task it is likely that ratings assigned by different", individuals 
will be congruent; and, because the scales help the raters 
fobus directly on actual job behavior examples instead of 
traits it is likely that greater differentiation between^ 

persons being. rated "will result (Borman and Dunnette/^. 561). 

/' 

The process (above) addresses salient performance dimensions. 
Faculty are used to construct BARS for evaluation of academic 
<f apartment chairpersons. A more detaixea elaboration of the 
method cbuld involve academic and support services adminis- 
trators. Such an elaboration most likely would yield a measure 
of validation to overall performance appraisal as well as 
provide several new dimensions for assessment* \ 

Meanings of response categories^ can be empirically verified. 

j 

Blood (1979/ p. 114) explains the double-elimination verification 

system as one where every behavioral episode generated is 

subjected to two * judgments in our empirical sample drawn 

from the rater population (faculty). Each item, he says, 

is judged as to the performance dimension it represents , and _ 

all items of low agreement are dropped. The remaining pool of 

scale anchors then, consists of only those items which have 



high commonality of meaning within the rater population. 
4. All of the BARS generated are expressed in the language of the 

rater since the raters have actually generated the scales. 
It appears that the methodology itself aids in clarification of goals, 



and by its participative and callaboratlve characteristics one could assume 

that the some of the basic values inherent in a collegial body are being taken 

\ 

into account. 

V 

Generation of Scales \ 1 

The chairpersons of 14 academic departments in a college of arts and ~ ^ 

i 

sciences were to be the target of the evaluation effort using the behaviorally 
anchored rating scales. 

-A total of 28 faculty Members, two from each department, were asked to 
identify evaluative demensi/ons for assessing chairperson performance. For 
each department, one of the faculty members selected to participate in the 
process had five or less years experience in the department, and the other 
faculty member had to have at, least 10 years experience ir. the department. 
These criteria were achieved in each department. At a group meeting a brief 

definition ^as^ identified for each dimension after considerable discussion. 

\ v 
This part of the process identified eleven performance dimensions .(see Table 1) . 

The faculty were then asked to provide critical/behavioral incident^ that 

would demonstrate poor, adequate, and good performance on each of the eleven 

-dimensions. Not all faculty provided three examples for each dimension. 

Instead of 924 (28 X 11 x 3) incidents, a total of 753 incidents were generated. 

A three-member review panel Examined the incidents and el" nated duplicate 

(redundant) incidents, non-behavioral episodes, and ambiguous episodes. This 

process resulted in a final count of 246 items. All of these items^ were 
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re-written into an "expectations" format. Each illustrative incident was 
stated in the fprm ."could be expected to*..," instead of remaining in a form 
whiohu-WOJulxi "Imply that the^ chairperson to be rated actually had to exhibit 
the specific behavior.. (Blood, 1979; Campbell, et. a.; Smith and. Kendall) . 

A total of 42 (three from each department) different faculty was asked 
to make two judgments for the list of items. The first judgment required 
categorization of the 246 items into the eleven performance dimensions. The 

second judgment required placement of each item on a 7-point scale based upon 

1 

the level of performance indicated. A rating of one represented the lowest 

I 

level of performance and a rating of seven represented the highest level of 

performance. Any item for which agreement as to dimension represented was 

\ x 

not at least 75 percent was eliminated from further consideration. This process 
of elimination forces a level of consensus on a final set of items. 

The grouping of the final pool of items into dimensions yielded only seven 
of the original eleven dimensions since there were not enough items for 
constructing. four scales \see Table 1). Blood (1979) says this circumstance 
can occur because: (1) some dimensions are not well-defined; (2) dimensions 

are similar to otners; or (3) there is simply a low level of agreement as to 

\ 

the appropriateness of the behavior as specified. 

Scales were then constructed for the remaining items. At least six items 
were used to anchor the meaning of scale points. The mean rating of the item 
located the item on the scale. By way of example, one of the scales is shown 
in Table 2. 

The seven .scales in their final ^form are to be distributed to all faculty 

\ \ 

of 14 academic departments of a collegeof arts and sciences in a state university 
Members of this faculty have participated in the generation of the rating 
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scales' as indicated by the procedure thSis outlined. 

r . . • , 

Results/Conclusions 

In a recent publication on evaluation .o£ administrative performance, 

Farmer (o. 18) points out that rating scales have several weaknesses although 

permitting ease of administration and anonymity. Weaknesses noted were biases 

. introduced by: (1) friendship, (2) quick guessing, (3) apF trance, (4) 
r \ 
prejudices, (5) halo effects, (6) errors of central tendency, and (7) leniency. 

The Jaehaviorally anchored scales proposed in this paper should respond 

positively in the elimination of most of the weaknesses identified. The method 

proppsed is similar to that developed by Findlay College by Rasmussen J197S) 

although Rasmussen grouped scale items for dimension identification with factor 

analysis. # 

The scales, as applied, are to serve as a primary information source in 

i 

a chairperson performance assessment system currently being implementsd. 
Application and examination/analysis of behaviorally anchored rating scale 

K 

.results, interviews with faculty and key administrative staff, and self-reports 
of chairpersons, will enable the college of arts and science administrators 
to thoroughly assess the performance of the chairpersons and the behavioral 
specifications will not only afford the chairpersons\eaningful feedback but 
may also indicate what kind of behavior should be demonstrated. Of course,"^ 
a set of scales could be developed from the administrator point of v:iew, as 
.well. The information base generated by the activities identified here should 
be of much assistance in goi| setting and in the definition of desired behavior. 
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Table 1 

* Dimensions for Evaluating Chairpersons 

*A+ General Administration - management of department office, including 
record keeping and clerical staff. 

*B. Resource Management - the manner and quality of allocating fiscal and 
other resources. 

*C. Sensitivity to Faculty - the extent to which faculty needs\are identified 
and addressed. V 

D. Planning - the demonstration of some better future and concept of what 
is desirable/ 

*E. Department Representation - the .extent to which the chairperson interacts 
with publics internal and external to the university. t /- 

*F. Communication with Faculty - the degree to which facult^axe-Jcept 

informed of organization policies, regulation's, plans, and procedures, 

G. Department Organization - the extent to which the faculty is deployed 
and managed to attain- department goals. 

H. Interactions with Students - the extent to which quality students are 
recruited and advised: general supervision of graduate students. 

I. Professionals Development - the degree of encouragement and support 
given to individual faculty, activity. 

*J.. Evaluation of Faculty - the extent to which various facets of faculty 
effort is .evaluated . 

*K. Curriculum Administration - the extent to which program features are 
1 monitored and modified. 
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*Dimensions identified by asterisk were retained in the final scales. 



v Table 2 
General Administration 



you would expect this chairperson to have developed a complete set of 
office procedures apd administrative forms 



you would expect this chairperson to maintain a set -of statistics 
° (information) about recruitment, attrition/, grades, and £>lacetnent 



of students 



\ 



you would expect this chairperson to have the ^lerical staff brought 
o up-to-date on the processing of departmental requests for supplies, 
- materials, etc. \ 



4- 



you would expect this chairperson to rely on a personal system of 
information storage creating dependency on part of office staff 

\ 

\ 

you would expect this chairperson to be confused about /the scheduling 
° of work to be done in the office ^ / 

- -X 

/ 

/ 

/ i 

you would expect this chairperson to be unable to locate important 
° student records, or a faculty member's request for travel funds 
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